The present invention relates to an improved motion compensation method. In particular, the present invention relates to a method for generating new image data based on motion compensated data of other images and to a corresponding motion compensator.
Motion compensation is employed in an increasing number of applications, in particular in digital signal processing of up-to-date television receivers. Specifically, modern television receivers perform a frame-rate conversion, especially in a form of a motion compensated up-conversion, in order to increase the picture quality of the reproduced images. Motion compensated up-conversion is performed, for instance, for video sequences having a field or frame rate of 50 Hz to higher frequencies like 60 Hz, 66.67 Hz, 75 Hz, 100 Hz, etc. While a 50 Hz input signal frequency mainly applies to television signals broadcast in accordance with PAL or SECAM standards, NTSC based video signals have a field rate of 60 Hz. A 60 Hz input video signal may be up-converted to higher frequencies like 72 Hz, 80 Hz, 90 Hz, 120 Hz, for example.
Frame rate conversion algorithms require the generation of intermediate images which reflect the video content at temporal positions different from that of the input video sequence of, for instance, 50 Hz or 60 Hz. During interpolation of image data of the input images, the motion of moving objects has to be taken into account in order to appropriately reflect changes of the image content caused by object motion.
The motion to be used during interpolation of the image data is determined based on the two closest images of the input video sequence. Generally, motion estimation is performed on a block basis. In order to avoid visible block structures in the motion compensated images, the motion vectors determined on a block basis are preferably assigned to each individual pixel wherein the resulting field of motion vectors is subjected to filtering.
Specifically, each image block receives a motion vector estimated on the basis of the input images. To avoid visible block structures in motion compensation, these blocks are filtered in a block erosion unit. The filter recursively divides in a multi-step processing each block into a number of sub-blocks and assigns a motion vector to each sub-block. After the last step of block erosion, a separate motion vector is assigned to each pixel position in the image to be interpolated. In accordance with the assumption that noticeable moving objects cover an area of several pixels and object motion is uniform, no vector assignment to the exact pixel position is required. Otherwise, a shift of the starting point of the motion vector towards the landing point in relation to the temporal position of the output image to the adjacent input images is necessary. In accordance with the received motion vector, motion compensation shifts the image data of the pixel referenced by the motion vector. The amount of shifting image data of the referenced images does not only depend on the length of the received motion vector, but further depends on the relative temporal position of the new image to be generated between the previous and subsequent images.
When generating new image data from two images, preferably the previous and subsequent images, the motion compensation algorithm references image data in the previous and the subsequent images. While the image data from the previous image is shifted forward in the motion vector's direction, the image data from the subsequent image is shifted backwards. The amount of forward and backward shift is determined in accordance with the temporal position of the new image with respect to the referenced images. While the shift of the forward shift is executed in accordance with a calculated fraction of the motion vector's length, the backward shift is performed with the complementary negative fraction of the motion vector. The referenced pixel values at each pixel position are interpolated. The interpolation can also be based on a weighing of the image data from the referenced images based on the temporal position with respect to the newly generated image.
The picture quality achieved by such interpolation algorithms is deteriorated if the interpolated images include non-active image areas. Such non-active image areas may be employed to adapt the image format of the active image area to the display format of a display device. In particular, letterbox type images employ black bars either at the top and bottom of the image or at the left and right side thereof in order to adapt a widescreen image for display on a conventional display format or vice versa.
Image areas next to such black bars of a letterbox type image suffer from missing motion vectors. Consequently, interpolation can only be based on a weighing of the image data from the two subsequent images without reference to the true motion of objects.
Such problems mainly occur when moving objects cross the border of the active image area, in other words, an object is either moving into the image or leaving the image such that the respective image content is only present in one of two subsequent images. Such motion can not be detected by conventional motion estimation algorithms. A promising new approach for determining motion vectors for the image borders of active image areas is the application of line vectors or column vectors which reflect the predominant motion within a line or column of blocks. A motion vector for image blocks, which include parts of moving objects which cross the image border and consequently have no counterpart in the other image, can be determined based on the motion vector of the respective line or column of blocks.
However, motion compensation based on line or column motion vectors result in an interpolation of image data from outside of the active image area. Consequently, inappropriate image data is introduced into the interpolation process. Due to interpolation, image data of the active image area are interpolated with the uniformly black image data causing a respectively impaired interpolation result.
For letterbox type images, this picture quality degradation is caused by motion vectors directed into the letterbox area. Pixels from the letterbox area are combined with pixels from the active image area such that the interpolated image data include black dots stemming from the letterbox area.